CONCURRENT DUAL PIPELINE FOR ACQUISITION, PROCESSING 
AND TRANSMISSION OF DIGITAL VIDEO AND HIGH RESOLUTION 
DIGITAL STILL PHOTOGRAPHS 

1 Technical Field 

2 The technical field relates to video imaging systems, and, in particular, to 

3 concurrent dual video and still image pipelines. 

4 Background 

5 Digital cameras are widely used to acquire high resolution still image 

6 photographs. Digital video cameras are also used to record home videos, television 

7 programs, movies, concerts, or sports events on a disk or DVD for storage or transmission 

8 through communications channels. Some commercial cameras are able to take both 

9 digital video and digital still image photographs. However, most of these cameras 

1 0 required a user to switch between a video recording mode and a digital still image mode. 

1 1 Separate pipelines are generally used for each of the video recording and still image 

1 2 modes. Examples of these cameras include SANYO ID-SHOT® and CANNON 

1 3 POWERSHOT S300®. The SANYO ID-SHOT® uses an optical disk, whereas the 

14 CANNON POWERSHOT S3 00® uses one of the synchronous dynamic random access 

1 5 memory (SDRAM). However, both products are still image cameras that have the 

1 6 capability of taking video clips, using separate pipelines. 

1 7 Other cameras use a single software pipeline to acquire both digital video and low 

1 8 quality still images, by taking one of the video frames as is, and storing the particular 

19 video frame as a still image. Examples of such cameras include JVC GR-DVL9800®, 

20 which is a digital video camera that allows a user to take a picture at certain point in time. 

2 1 However, the pictures taken generally are of low quality, because a low resolution video 

22 pipeline is used to generate the still image pictures. 

23 Summary 

24 A method for concurrently acquiring, processing, and transmitting digital video 

25 and still images includes acquiring video frames from one or more image sensors, and 

26 processing the video frames using a video pipeline. The video pipeline may include one 

27 or more processors. The method further includes temporarily storing the video frames in 

28 a frame buffer when one or more high resolution still images are acquired during the 

29 video frame acquisition, and processing the high resolution still images using a still image 

30 pipeline. The still image pipeline may run concurrently with the video pipeline. 
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1 As a result, high resolution still image photographs may be acquired during video 

2 recording without any interference with the video recording or any impact on the quality 

3 of the video or the still image photographs. 

4 Description of the Drawings 

5 The preferred embodiments of the concurrent dual video and still image pipeline 

6 will be described in detail with reference to the following figures, in which like numerals 

7 refer to like elements, and wherein: 

8 Figure 1 illustrates an exemplary operation of an exemplary concurrent dual video 

9 and still image pipeline; 

10 Figure 2 illustrates a preferred embodiment of a video camera system using the 

1 1 exemplary concurrent dual video and still image pipeline of Figure 1; 

12 Figure 3 illustrates an exemplary hardware implementation of the exemplary 

1 3 concurrent dual video and still image pipeline of Figure 1 ; 

14 Figures 4 A- 4C are flow charts describing in general the exemplary concurrent 

15 dual video and still image pipeline of Figure 1; and 

16 Figure 5 illustrates an alternative embodiment of the exemplary concurrent dual 

17 video and still image pipeline of Figure 1 . 

1 8 Detailed Description 

1 9 A digital video camera system may utilize a concurrent dual video and still image 

20 pipeline that simultaneously acquires, processes,, transmits and/or stores digital video and 

21 high resolution digital still image photographs. The concurrent dual pipeline may include 

22 a video pipeline optimized for digital video and a still image pipeline optimized for high 

23 resolution digital still images. As a result, high resolution still image photographs may be 

24 acquired during video recording without any interference with the video recording or any 

25 impact on the quality of the video or the still image photographs. 

26 Figure 1 illustrates an exemplary operation of an exemplary concurrent dual video 



27 and still image pipeline, which is capable of simultaneously delivering digital video 

28 frames 120 and high resolution digital still images 110. The video frames 120 may be 

29 acquired at, for example, 30 frames per second (fps). During video frame acquisition, a 

30 snapshot 102 may be taken to acquire a particular still image 1 10 in high resolution. 

3 1 During the high resolution still image acquisition, all other video frames 120 may be 

32 temporarily stored, i.e., buffered, in a frame buffer 330 (shown in Figure 3). Both the 

33 video frames 120 and the high resolution still image 1 1 0 may be transmitted through 

34 communications channels, such as a network. 
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1 Figure 2 illustrates a preferred embodiment of a video camera system 200 using 

2 the exemplary concurrent dual video and still image pipeline. In this embodiment, a 

3 video pipeline 220 and a still image pipeline 210 share a same high resolution image 

4 sensor 240. The high resolution image sensor 240, which may be a charge coupled 

5 device (CCD) sensor or a complimentary metal oxide semiconductor (COMS) sensor, 

6 may need to take high resolution still images 110 while acquiring medium resolution 

7 video frames 120. This embodiment is relatively inexpensive, because the video camera 

8 system 200 utilizes only one hardware processing pipeline 300 (shown in Figure 3) with 

9 one image sensor 240 and one processor 360 (shown in Figure 3). 

1 0 The image sensor 240 typically continuously acquires high resolution video 

1 1 frames 120 at a rate of, for example, 30 fps in the United States. Each of the high 

12 resolution video frames 120 may possess enough information to generate a high 

13 resolution still image photograph 110. When a user is not interested in taking a high 

14 resolution still image photograph 110, the only pipeline that is running may be the 

15 medium resolution (for example, 720x480) video pipeline 220. When the user wants to 

1 6 acquire a high resolution still image 1 1 0, the image acquired by the high resolution image 

17 sensor 240 at that particular time may be used both in the video pipeline 220 as well as in 

18 the high resolution still image pipeline 210 (described in detail later). 

19 The video camera system 200 may include a storage device 250 and a connection 

20 with a communications channel/network 260, such as the Internet or other type of 

21 computer or telephone networks. The storage device 250 may include a hard disk drive, 

22 floppy disk drive, CD-ROM drive, or other types of non- volatile data storage, and may 

23 correspond with various databases or other resources. After the video frames 120 and the 

24 high resolution still images 1 10 are acquired, the video frames 120 and the high 

25 resolution still images 110 may be stored in the storage device 250 or transmitted through 

26 the communication channel 260. 

27 Figure 3 illustrates an exemplary hardware implementation of the preferred 

28 embodiment of the exemplary concurrent dual video and still image pipeline. This 

29 embodiment includes the single hardware processing pipeline 300 supporting two 

30 software pipelines. A sensor controller 310 may be controlled by a user to retrieve high 

3 1 resolution mosaiced still images 1 1 0 at a rate of, for example, one every thirtieth of a 

32 second to generate a video signal. The sensor controller 310 may then store the selected 

33 high resolution still images 110 into a memory 320. The memory 320 may include 

34 random access memory (RAM) or similar types of memory. Next, the high resolution 



HP 10019005 



3 



1 still images 1 1 0 may be processed using the processor 360, which may be a 

2 microprocessor 362, an ASIC 364, or a digital signal processor 366. The ASIC 362 

3 performs algorithms quickly, but is typically application specific and only performs a 

4 specific algorithm. On the other hand, the microprocessor 362 or the digital signal 

5 processor 366 may perform many other tasks. The processor 360 may execute 

6 information stored in the memory 320 or the storage device 250, or information received 

7 from the Internet or other network 260. The digital video and still image data may be 

8 copied to various components of the pipeline 300 over a data bus 370. 

9 In the video pipeline 220, the processor 360 may downsample, demosaic, and 

10 color correct the video frames 120. Next, the processor 360 may compress and transmit 

1 1 the video frames 120 through an input/output (I/O) unit 340. Alternatively, the video 

12 frames 120 may be stored in the storage device 250. 

13 Both pipelines may be executed concurrently, i.e., acquiring high resolution still 

14 image photographs 1 10 during video recording. A frame buffer 330 may store video 

15 frames 120 while the single processor 360 is processing the high resolution still image 

16 110. The sensor controller 3 1 0 may still capture video frames 1 20 at a rate of, for 

17 example, 30 fps, and store the video frames 120 into the memory 320. The processor 360 

18 may need to downsample the video frames 120 and send the downsampled video frames 

19 120 into the frame buffer 330. The frame buffer 330 may store the downsampled video 

20 frames 120 temporarily without further processing, incurring some delay in the video 

21 pipeline 220. However, this delay may be compensated by a similar buffer on the 

22 receiver end (described later). During video frame buffering, the high resolution still 

23 image 110 may be processed by the processor 360, using complex algorithms. At the 

24 same time, the video frames 120 are continuously stored into the memory 320, 

25 downsampled, and sent into the frame buffer 330 to be stored. 

26 During the time the processor 360 processes the high resolution still image 1 1 0, 

27 the frame buffer 330 is being filled. Once the high resolution still image 110 has been 

28 processed, transmitted and/or stored, the video pipeline 220 takes over and starts 

29 processing the video frames 120 stored and accumulated in the frame buffer 330. 

30 Thereafter, the video pipeline 220 transmits the video frames 120 into the I/O unit 340 or 

3 1 the storage device 250 in order to empty the frame buffer 330. At the same time, new 

32 video frames 120 may be acquired and stored into the memory 320. Accordingly, the 

33 video pipeline 220 may need to clear the frame buffer 330 faster than the speed the video 

34 frames 120 are acquired. After the frame buffer 330 is emptied, another high resolution 
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1 still image 110 may be taken. The frequency of high resolution still image 1 1 0 that can 

2 be acquired depends on the capacity of the processor 360 and the size of the frame buffer 

3 330. 

4 If the video frames 120 are stored, the above operation may not affect the overall 

5 video camera system 200, since a delay in storing may not be noticeable. On the other 

6 hand, if the video frames 120 are transmitted, the delay may be noticed. However, all 

7 video streaming solutions have another frame buffer on the receiver end, which may 

8 absorb the delay, so the viewer at the other end of the communications channel may not 

9 notice the delay. 

1 0 Alternatively, the high resolution still image 110 may be acquired and stored 

1 1 without any processing. During this procedure, some video frames 120 may be stored in 

12 the frame buffer 330 to move the large amount of unprocessed pixels across the data bus 

13 370 into the storage device 250. In this embodiment, the frame buffer 330 may be much 

14 smaller, and the delay in video transmission may be smaller. However, the high 

15 resolution still images 110 may not be available until the video acquisition stops and 

1 6 allows the video camera system 200 to process all the high resolution still images 110 that 

17 have been acquired during the video recording. Accordingly, any image transmission that 

1 8 the user desires may be delayed. The above described solutions are for illustration only, 

1 9 and one skilled in the art will appreciate that many other intermediate solutions will be 

20 possible with this architecture for the concurrent dual video and still image pipeline. 

21 Although the video camera system 200 is depicted with various components, one 

22 skilled in the art will appreciate that the video camera system 200 can contain additional 

23 or different components. In addition, although aspects of an implementation consistent 

24 with the present invention are described as being stored in memory, one skilled in the art 

25 will appreciate that these aspects can also be stored on or read from other types of 

26 computer program products or computer-readable media, such as secondary storage 

27 devices, including hard disks, floppy disks, or CD-ROM; a carrier wave from the Internet 

28 or other network; or other forms of RAM or ROM. The computer-readable media may 

29 include instructions for controlling the video camera system 200 to perform a particular 

30 method. 

3 1 Figures 4A- 4C are flow charts describing in general the exemplary concurrent 

32 dual video and still image pipeline. Referring to Figure 4 A, operation of the video 

33 pipeline 220, shown on the left, typically results in continuous processing of video frames 

34 120. Operation of the still image pipeline 210, shown on the right, typically results in 



HP 10019005 



5 



1 processing a high resolution still image 1 10 every time the user wants to acquire a high 

2 resolution photograph 110. 

3 After raw pixel video data of video frames 120 are acquired, for example, at 

4 1 024x1 008 and 30 fps (block 400), the video frames 120 may be downsampled and 

5 demosaiced in order to save memory space (block 410). Then, the frame buffer 330 may 

6 buffer the video frames 120 while the high resolution still image 1 10 is being acquired, 

7 processed, stored, and/or transmitted (block 420). Alternatively, demosaicing may be 

8 performed after the video frames 120 are buffered. Thereafter, the video pipeline 220 

9 may start emptying the frame buffer 330 as fast as possible. Once the frame buffer 330 is 

10 emptied, another high resolution still image 110 may be acquired. Next, color correction 

1 1 may be performed (block 430), followed by compression of the video frames 120 (block 

12 440). Finally, the video frames 120 may be stored and/or transmitted through 

1 3 communications channels 260 (block 450). 

14 For high resolution still images 110, sophisticated demosaicing may be performed 

1 5 (block 412), followed by high quality color correction (block 432). The high resolution 

1 6 still images 1 1 0 may optionally be compressed (block 442), and then stored and/or 

1 7 transmitted through similar communications channels 260 (block 452). 

1 8 Figure 4B illustrates in detail the operation of the high resolution still image 

1 9 pipeline 210. The sophisticated demosaicing process (block 412) utilizes a high quality 

20 demosaicing algorithm that generates a high quality color image from the originally 

21 mosaiced image acquired by the image sensor 240. Examples of the demosaicing 

22 algorithm are described, for example, in "GENERALIZED WIENER 

23 RECONSTRUCTION OF IMAGES FROM COLOUR SENSOR DATA USING A 

24 SCALE INVARIANT PRIOR" by David Taubman, Proceedings of the ICIP2000, 

25 published September 10, 2000, which is incorporated herein by reference. The 

26 demosaicing process is a time consuming filtering operation, which may gamma-correct 

27 the input if the image sensor 240 has not done so, resulting in excellent color image 

28 quality with almost no demosaicing artifacts. For example, demosaicing for high 

29 resolution still images 110 may filter the original image with a 10x10 linear filter. The 

30 algorithm takes into account the lens used for acquisition, as well as the spectral 

3 1 sensitivity of each of the color filters on the mosaic. 

32 Once the high resolution still image 1 10 is demosaiced, the high resolution still 

33 image 1 1 0 may be color corrected depending on the illumination present at the time of the 

34 capture (block 432). Complex transformation matrixes may be involved to restore 
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1 accurate color to the high resolution still images 1 1 0, in order to generate an excellent 

2 photograph. The color correction algorithms, may be similar to the algorithm used in the 

3 HP -PHOTOSMART 618®. 

4 Figure 4C illustrates in detail the operation of the video pipeline 220. A high 

5 quality video pipeline 220 may be very demanding in terms of computation. Because the 

6 video processing needs to be achieved at, for example, 30 fps, downsampling may be fast. 

7 In addition, lower resolution video frames 120 (for example, 720x480 pixels) demands 

8 much less quality demosaicing (block 410), because the human visual system may not 

9 notice certain artifacts at high video processing rates. For example, demosaicing for 

1 0 video frames 1 20 may filter the original image with a 4x4 linear filter. Similarly, color 

1 1 correction may be simpler because high quality is not needed on the video side (block 

12 430). 

1 3 Figure 5 illustrates an alternative embodiment of a video camera system 500 using 

14 the exemplary concurrent dual video and still image pipeline. In this embodiment, two 

1 5 separate hardware processing pipelines may run concurrently within the video camera 

16 system 500, using two separate image sensors 540, 542. For the video processing 

1 7 pipeline 220, a medium resolution sensor 540 (for example, 720x480 pixels) is used, 

1 8 whereas a high resolution sensor 542 (for example, 2000x2000 pixels) is used for the high 

1 9 resolution still image pipeline 210. With separate hardware processing pipelines, no 

20 interference exists, therefore no large frame buffer is needed. Even though presenting an 

21 easier solution, this alternative embodiment may be more expensive, because duplicate 

22 hardware are involved while the two hardware processing pipelines may be used 

23 simultaneously only occasionally. Alternatively, the two separate hardware processing 

24 pipelines may use the same image sensor (not shown). One skilled in the art will 

25 appreciate that many other intermediate solutions will be possible with this architecture 

26 for the concurrent dual video and still image pipeline. 

27 While the concurrent dual video and still image pipeline has been described in 

28 connection with exemplary embodiments, those skilled in the art will understand that 

29 many modifications in light of these teachings are possible, and this application is 

30 intended to cover any variation thereof. 
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